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Abstract 

This project looks to improve first responder 
suuational awareness - using"" me 
techniques . The prototype system combines wireless 
communication, real-time location determination, digital 
imaging, and three-dimensional graphics. Responder 
locations are tracked in an outdoor environment via GPS 
and uploaded to a central server via GPRS or an 802.11 
network Responders can also wirelessly share digital 
images and text reports, both with other responders and 
with the incident commander . A pre-built three 
dimensional graphics model of the emergency scene is 
used to visualize responder and report locations . 
Responders have a choice of information end points, 
ranging from programmable cellular phones to tablet 
computers . The system also employs location-aware 
computing to make responders aware of particular 
hazards as they approach them. The prototype was 
developed in conjunction with the NASA Ames Disaster- 
Assistance and Rescue Team and has undergone field 
testing during responder exercises at NASA Ames . 

Keywords: first responders, emergency management, 
disaster management, pervasive computing, GPRS, GPS, 
situational awareness 


1. Introduction 

First responders routinely face dangerous 
environments and situations, ones ranging from fires to 
natural disasters to terrorist attacks. Successfully 


conducting operations in these environments requires 
rigorous training, properly chosen technology, strong 
incident- command, and an appropriately managed flow of 
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places, Chapter 9 of the 9/1 1 Commission Report [1]. 

Responders at an emergency scene need answers to 
questions: Where am I? What do we know about the 
scene? What hazards exist? Where are other responders 
located? Where are the victims? The hope is that by 
providing appropriate and timely answers to these kinds 
of questions, first responders will be better prepared to 
respond to and manage emergencies. Of course, the 
appropriateness of an answer will almost certainly vary 
by responder role, and there is little question that too 
much information is just as harmful as too little. 

This paper describes a prototype system, the Smart 
Systems Research Laboratory (SSRL) Responder Tool, 
that was built to examine innovative ways of increasing 
first responder situational awareness and thus their ability 
to respond to the crisis at hand. The system makes use of 
off-the-shelf hardware, including tablet computers, 
personal digital assistants (PDAs), and programmable 
cellular phones. System software was built on Microsoft’s 
.NET platform and brings together web services, wireless 
communication, and computer graphics. The system 
allows^ for report and image exchange and archiving, as 
well as real-time location tracking of individual 
responders via the” Global Positioning System (GPS). 
Responder and report locations are visualized in real-time 
using three-dimensional computer graphics. The tool 
grew out of earlier work done at SSRL related to space 
mission training [2, 3], 

Responders often use the term “C3” to describe the 
response to a disaster scene: command, control, and 
communication. The SSRL Responder Tool is intended as 
an aid for both the command and communication portions 
of C3. It is intended to bring a higher level of consistency 
to the interaction between different first response teams. 



The remainder of this paper is organized as follows. 
First, background on the problem and the motivation to 
study it are presented. The methods used in constructing 
the system, including software and hardware 
architectures, are described, as are results obtained with 
the system. Finally, conclusions and avenues for future 
work are presented. 

2. Background 

The NASA Ames Research Center is home to an all- 
hazard federal emergency response and recovery team 
known as the Disaster Assistance and Rescue Team 
(DART) [4]. This team of approximately two hundred 
and fifty mainly volunteer members has extensive training 
facilities at NASA Ames, including a large Collapsed 
Structure Rescue Training Site (shown’ in Figure 1). In 
late 2003, DART leaders decided to reach out to 
engineering and science groups at NASA Ames, other 
federal facilities, and. Silicon. Y alley...in. an effort to i n fuse, 
advanced technologies into the DART emergency scene 
workflow. This ongoing program has led to DART 
members giving guidance and advice to technology 
groups as they strive to solve emergency responder 
issues. It has also resulted in periodic field tests and 
technology demonstrations. For more on one such 
demonstration, see [5], 



Figure 1 . Photo of the NASA Ames Collapsed 
Structure Rescue Training Site. 


As part of this program, members of SSRL at NASA 
Ames initially set to work on the specific problem of 
providing information technology aids to assist the 
construction of wooden building shores. This problem, 
and the prototype system designed to address it, grew into 
one that addressed the broader issue of emergency scene 
situational awareness for all responders, including those 
at the incident command post. 

Before conducting search and rescue operations in a 
building that has collapsed (due perhaps to an earthquake, 
hurricane, or fire), specialized responders may need to 
construct wooden shores for the structure. These shores 
are used to stabilize the structure, thereby reducing the 


risks to responders of operating in and around it The 
process of constructing the shore involves one set of 
responders communicating a desired shore type and 
dimensions to another set of responders responsible for 
cutting wood [6]. The cutting team may be at a 
centralized location away from the structure, so 
communication between the two team elements may be 
via radio or via slips of paper sent back and forth by 
runners. Once wooden pieces of the shore are cut, runners 
ferry them to the shore site. 

In striving to improve this workflow, SSRL members 
looked to combine wireless communication, small form 
factor computing, information management, and 
computer graphics in an effort to increase overall 
situational awareness. With the addition of a fifth element 
(real-time location determination), these information 
technologies can further be used to aid general situational 
awareness of an emergency scene. 
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Several areas of research and commercial enterprise 
are relevant to the work described in this paper. Certainly 
the five areas mentioned in the previous section—!) 
wireless communication, 2) small form factor computing, 
3) information management, 4) computer graphics, and 5) 
location determination — -have vast bodies of research 
associated with them. The COORDINATORS effort of 
Wagner et al. has a similar goal to the work reported here 
[7]. Their work looks to provide decision support to first 
responders by reasoning about who should be doing what 
and when. It makes use of PDAs, wireless 
communications, and a Honeywell proprietary asset 
location technology to track responders, victims, and 
equipment. Differences < between the two include the 
SSRL Responder TooPs use of 3D graphics, support for 
shoring operations, and emphasis on Smartphones. 

One of the appliances used in the work reported here is 
a programmable cellular phone. Recent work by Roussos 
et al. outlines general challenges associated with using 
Smartphones as Information End Points (IEPs) [8]. 
Although it does not specifically examine first responder 
use, it contains many relevant lessons learned by the 
authors. Kun et al. and Miller et al. provide good 
descriptions of the challenges of integrating information 
appliances into first responder workflows and ensuring 
interoperability [9, 10]. Sawyer et al. describe techniques 
for providing secure mobile access to information for first 
responders [11]. 

Patterson et al. discuss challenges associated with 
location-aware computing [12]. They also describe a 
futuristic scenario where robots use location-aware 
computing to respond to a collapsed structure disaster. 
Kwan and Lee look at the applicability of location-aware 
computing to improve emergency response [13]. Like this 



work, they describe the fusion of real-time location 
information with three dimensional computer graphics. 
The TellMaris project made use of portable computer 
graphics and GPS data and applied it to a maritime 
application [14]. 

Location determination is a requirement for many 
different applications. Probably the best known real-time 
location system (RTLS) is GPS [15]. It is mostly intended 
for outdoor use, insomuch as the transmitted satellite 
signals penetrate structures poorly. The Federal 
Communication Commission’s E911 mandate [16], 
among other things, has prompted research and 
development in metropolitan and indoor location 
determination techniques, including ones based on 
television signals [17]. For a good overview of current 
techniques and challenges, see [18]. 

4. Methods 

— The S SRU-Respoade r Tool -is divided into thre e- 

distinct components: 

1) Responder components; 

2) Command Post components; 

3) Server components. 

This section of the paper looks at the architecture, 
hardware, and software of all three. 

4.1 Responder Architecture 

Responder components are illustrated schematically in 
Figure 2. Responders carry Bluetooth GPS receivers with 
them, mounted on their helmets. Both Hewlett Packard 
iPAQ and Garmin GPS 10 Bluetooth receivers have been 
tested, with the latter device offering Wide Area 
Augmentation System (WAAS) capabilities for greater 
positioning accuracy. 

Responders have a choice of IEPs when using the 
system. Multiple IEP types were incorporated to 
accommodate different responder roles. Some responders 
are able to make use of PDAs and small tablets while still 
performing their responder duties. For other responders, a 
device on the size of a cellular phone is all that can be 
accommodated. Various responder IEPs used in the SSRL 
Responder Tool are shown in Figure 3, 


BT GPS Device 



Figure 2. Overview of the responder components. 

The smallest IEP is a programmable cellular phone, an 
Audiovox SMT 5600 running Windows Mobile 2003 
Second Edition on the Cingular network. Using the 
phone, responders can send and receive images and text 
reports via General Packet Radio Service (GPRS). 
Location data is read from the GPS receiver via Bluetooth 
and also uploaded to the server via GPRS, thereby 
allowing the system to track responder locations in real- 
time. Finally, responders can browse portions of their 
Field Operations Guide on the phone (as they can on all 
IEPs). 



Figure 3. Responder IEPs. From left to right are the 
Smartphone, tablet, and PDA. All are running the 
SSRL Responder Tool. Ruler units are inches. 

The second responder IEP is a PDA. At present, its 
capabilities are similar to that of the cellular phone, with 
the exception that it communicates via a local 802.1 lb/g 
network instead of GPRS. A combination of Hewlett 
Packard iPAQs and Dell Axims has been tested, again 
running Whndows blob lie 2003 Second Edition. 

The final IEP is a small-form-factor tablet computer. 
Devices that have been tested include a Toshiba Portege 
M200 and a Sony Vaio U71P touch-sensitive handtop. 
Since even PDAs are potentially too large for many 
responders to carry, some explanation of why tablets are 
offered is in order. The principal reasons were because 
they run full versions of Windows XP (thereby easing 


software development) and because they offer hardware 
accelerated graphics. These devices are the only IEPs in 
the current system that offer responders a three-, 
dimensional virtual environment of the emergency scene 
(although future work will look to add this feature to the 
PDAs). The virtual environment is also used to visualize 
a responder’s current location and the location of other 
responders, as well as locations where reports were made. 
Figure 4 shows a screen shot of a responder tablet 



Figure 4. Screen shot of a responder tablet. The 
upper panels deal with text reports. Images 
associated with reports, along with the Field 
Operations Guide, can be shown in the lower left 
panel: The lower right panel displays a virtual 
environment of the scene, with the responder’s 
current location indicated by the icon. The virtual 
environment models the NASA Ames Collapsed 
Structure Rescue Training Site shown in Figure 1. 

In an effort to reduce hardware costs, the current 
prototype does not use hardened IEPs. A deployed system 
would unquestionably utilize devices that were resistant 
to dust, heat, moisture, and impact. Such devices are 
increasingly common and affordable. For example, 
Panasonic offers a hardened line of notebooks (known as 
Toughbooks) and non-hardened PDAs can be encased in 
armor (e.g., the OtterBox line of Armor from Otter 
Products). 

4.2 Command Post and Server Architectures 

The other two components of the SSRL Responder 
Tool are the Command Post (GP) and server portions, 
illustrated in Figure 5. The CP portion includes software 
and hardware for the Incident Commander (IC) and other 
staff to monitor reports, imagery, and responder locations, 
as well as possibly using the tool for overall emergency 
response coordination and management. It also includes 
an 802.1 lb/g wireless communications infrastructure, 
intended to cover not only the CP but also the local area 


where responders are working. The IC can use any of the 
responder IEPs to interact with the system. However, if 
the IC is fairly stationary, (or perhaps has a command 
vehicle available), a conventional notebook computer can 
be used. 



Figure 5. Ove r v i ew ~ of the Cor TTmaTid~~Post and 
server portions of the system . 


Wireless communication is a critical aspect of the 
SSRL Responder Tool. The power of the tool lies in large 
part with, the ability to collect and share information 
amongst responders and the IC. The various responder 
IEPs need network connectivity, which in the current 
prototype has been delivered either via a fieid-deployable 
802,1 Ib/g network or GPRS. At least in metropolitan 
areas, GPRS is reasonably ubiquitous, although it offers 
relatively little bandwidth. In the event that the cellular 
network is unavailable, a local 802.1 lb/g network can at 
least give some coverage to the responders and IC. It 
requires the deployment of a combination of wireless 
routers, access points, and range extenders (and making 
sure these devices are powered, either from batteries or 
generators). A final component of the communication 
infrastructure can include an optional satellite data link. 

The server is the communications hub. It stores text 
reports, images, and responder locations in a database. 
The server accepts and delivers information via an XML 
web service. It can either be collocated at the CP or, if a 
satellite link is available, remain at a remote facility (a 
local mirror would reside at the CP in case the satellite 
link fails). This configuration, with the server resident at a 
remote NASA facility and a satellite link bridging the 
local network, has been tested during a disaster exercise. 
The current system uses a single Dell Inspiron 8200 
running Windows Server 2003 as the server. 

4.3 Software 

Custom software was developed for all three 
components of the SSRL Responder Tool (meaning the 
responder IEPs, the CP computers, and the server). 



Microsoft’s .NET platfonn was used as the 
implementation platform and all of the code, including 
the graphics code, was written in C#. • 

The desire to- cater to a wide variety of different 
computer platforms (i.e., cellular phones, PDAs, tablets, 
servers) with maximum code sharing between the 
platforms made either J2EE/J2ME or .NET the natural 
implementation choices. The project could have 
unquestionably used either one. That said, we have been 
impressed with the .NET platform and its supporting 
software development kits. The SSRL Responder Tool 
makes use of, among other things, ADO.NET, managed 
Direct3D, and the .NET Compact Framework. They have 
made it easy to rapidly move ideas from design concept to 
working prototype. 

The major unit of data in the system is a Report A 
report can contain free-form text comments, shore 
dimensions (recalling that the tooTs origin was to assist 
with the construction of wooden building shores), a 
digital image, and a location. Responders can create and 
share these reports on the various IEPs. Reports are 
pushed to and pulled from the server via an XML web 
service published by the server. Reports are archived in a 
database on -the server and are made available in 
chronological order to all the responders on a team and 
the IC. Figure d shows aspects of the report user interface 
on the Smartphone and PDA. 



Figure 6 , Part of the report user interface on the 
Smartphone and PDA responder IEPs. On the 
Smartphone, the user has selected the image 
associated with a report (created during one of the 
field tests) and is viewing it. On the PDA, the user is 
viewing the chronological list of reports. 


Digital images associated with a report can assist the 
cutting crews and the IC in assessing a particular shore 
site. Digital images can either be acquired directly by the 
IEP or via an additional camera. The disadvantage of 
using an additional camera is that it is yet another device 
that consumes batteries and has to be hardened to work in 
a real disaster environment. The advantage is that, at least 
for now, such cameras have bettef optics than the cameras 
built into cellular phones. 

The location from which a report is uploaded is 
captured and associated with the report. Latitude and 
longitude are read from the GPS receivers, although these 
are converted to Universal Transverse Mercator (UTM) 
coordinates [19] in software in order to facilitate use of 
the coordinates in the virtual environment. The upload 
location of a report and the real-time location of 
responders are marked with icons in the virtual 
environment of the disaster scene. The virtual 
environment is available to staff at the CP and responders 
with tablet IEPs. ‘ 

The virtual environment uses pre-sp'eeified models of a 
particular disaster scene. That is* the mesh geometry is 
not determined in real-time at a disaster scene but instead 
in advance, through a combination of examining 
blueprints, CAD models (if available), and pictures. The 
mesh is stored in DirectX format in units of meters. A 
transformation matrix stored with the mesh aligns it with 
the local UTM grid, so positive z in the model 
corresponds to north and positive X to east. 

Unquestionably we would prefer to use graphical 
models generated in real-time at a disaster scene. Such 
models, if they could be generated quickly and accurately, 
would have the advantage of reflecting the scene as it 
truly is (e.g., the partial collapse of a structure could be 
captured). Pre-modeling', on the other hand, has the 
advantage of being able to capture information that would 
be hard or impossible for any real-time technique (say 3D 
photogrammetry) to obtain, such as the location of buried 
gas or electrical lines. The expense of pre-modeling a 
scene with enough fidelity to be useful to responders may 
be justifiable for high-value or high-risk assets (e.g., 
nuclear power facilities, refineries, prominent commercial 
and government buildings, mass transit infrastructure, 
etc.). The current prototype uses a single pre-modeled 
scene of the NASA facility shown in Figure 1. 
Constructing the scene took approximately 20 hours of 
work by a skilled 3D graphics modeler. 

The decision to use a web service to move data 
between computers was largely driven by the ease with 
which they can be implemented and consumed in the 
.NET environment. Experiments with a custom TCP 
protocol hovered between difficult and impossible on a 
real GPRS -based provider network. While such a 
protocol was possible, actually getting it to work would 
have almost certainly required assistance from the carrier. 



Conversely, consuming a web service over carrier 
networks under Windows Mobile is straightforward. The 
disadvantage of the web service is that it has 
architecturally required responder lEPs to poll for new 
reports instead of having the server push new reports only 
when and if they become available. 

The current prototype does not directly address 
security issues. Data is stored and transmitted as clear text 
and there is no user authentication (via passwords or 
biometrics). Real-time responder locations and report data 
are likely to be sensitive. For example, reports may 
include personal information about victims. While the 
current prototype has been architected to allow for 
eventual insertion of certain security-related features, a 
deployed system would need a more comprehensive 
security solution. 

5. Results 

Initial design work on the SSRL Responder Tool was 
done in November 2003. The first substantial field test 
occurred in May 2004 at the NASA Ames Technologist- 
Meets-Responder Exercise [5]. The system was 
demonstrated and used as part of the exercise scenario by 
engineers who had worked on the prototype (as opposed 
to responders). Responders and ICs that observed the 
system made many useful comments on how the tool 
could be refined and expanded. For example, several 
responders saw it as a useful tool for enhancing 
situational awareness by giving easier access to critical 
information during in-field briefings to relief teams. 
Before going into a structure or helping with the 
construction of a shore, a squad leader could use the 
reports, images, and virtual environment to help orient 
squad members and alert them to potential dangers. Other 
responders felt that the tooPs archiving capability would 
be useful for after-action reports and for developing 
training scenarios. 

Responders also expressed several concerns. Any 
device that consumes batteries is always a potential 
problem in the field — one responder remarked that no 
matter how advanced the tool* he would never leave his 
pencil and notebook behind. Concerns were also raised 
about the practicality of setting up a local 802.1 1 b/g 
network in the field. For this exercise,’ we did deploy a. 
network in the field, powering the access points off a 
generator, although even with multiple access points there 
were areas in and around the structure of inadequate 
signal strength. 

Beyond routine field testing, two other substantial 
field tests have been performed using the SSRL 
Responder Tool. It was used during a responder 
workshop in April of 2005, where the various 
components of the tool were once again operated by 
engineers (although this time in close cooperation with 


individual responders). A third field test in June 2005 
demonstrated key features of the tool to NASA and 
Department of Homeland Security representatives. 

6. Conclusions and Future Work 

There seems little doubt that technology aids intended 
to increase situational awareness will become more and 
more common at disaster scenes. The process of 
accepting a system should — and hopefully will — be 
rigorous. Only those tools that can definitively show a 
significant increase in overall situational awareness 
should be accepted. 

For fire and urban search and rescue (USAR) teams, 
accurate location determination within structures is of 
vital importance. As has already been mentioned , this is a 
very active area of research and development. When 
fused together with other modalities, such as GPS, this 
perhaps more than anything will bring immediate and real 
situational awareness benefits to both responders and 
commanders. 

Low-cost and robust field-deployable wireless 
networks are needed. In the event that metropolitan 
networks are down, responders would benefit by being 
able to deploy their own system, even if the range was 
relatively limited. Base stations could be mounted 
permanently on vehicles with some additional access 
points and range extenders deployed in a working area by 
responders. When the work area shifts, the network could 
be shifted with it. 

With the power of networking comes the associated 
risk of information overload. Careful attention will have 
to be paid to filtering the information presented to 
individual responders and to commanders. The SSRL 
Responder Tool has taken preliminary steps in this 
direction by delivering location-specific content (LSC) to 
responders via their lEPs. At present, the lEPs are pre- 
loaded with information about points of potential hazard 
or interest, no different from many off-the-shelf GPS 
devices. When a responder moves within a certain 
distance of the trigger point, the IEP flashes and issues an 
audible warning that LSC is available. Future work will 
look to deliver real-time over-the-air (OTA) LSC. For 
example, when approaching the entrance to a structure, 
the IEP will present the responder with an electronic 
version of any Building Marking System (BMS) [6] data 
associated with the building. BMS marks are currently 
made with orange spray paint on the sides of buildings 
and denote such things as the building address, structure 
and hazards evaluations, victim locations, search 
assessment marks, and team identifiers. Although BMS 
markings are standardized, interpreting marks and mark 
updates made by different teams can be a challenge at a 
long-duration disaster scene. 



Future work will also continue to address the specific 
problem of shoring. Shoring dimensions entered into the 
tool will be used to compute wood cut lengths. The 3D 
graphics capabilities of the PDA IEPs will be used to 
provide a dimensionally-accurate real-time visualization 
of the shore being constructed, along with the proper 
assembly sequence. Once complete, the SSRL Responder 
Tool will hopefully achieve a limited deployment with 
structure specialists at NASA Ames DART, growing into 
a larger deployment with other team elements. 

More work is needed on the human-computer interface 
issues associated with responders and computers. 
Smartphones and PDAs have the advantage of being 
inexpensive and easy to develop software for, but they 
may not represent the best interface appliance. Small 
screens can be difficult to read under the best 
circumstances; a stressful situation, or even bright 
sunlight, can greatly increase the difficulty. Operating 
small devices with gloves on is challenging, meaning 
— robusft-hands-free techniques - for'intemctrn^wfth IEPs'" 
would be welcome. Making effective use of heads-up 
displays (HUDs) is an interesting avenue for future 
research. 
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